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ABSTRACT 

This paper examines the design and findings of a 
survey of 31 faculty and 550 students participating in 7 
Annenberg/CPE New Pathways to a Degree electronic learning projects. 
Each project was selected because it proposed innovative ways to use 
technology to create new educational opportunities for "new majority 
learners" (students unable to attend college in conventional ways). 
Ail New Pathways courses used technologies in combinations, and 
communication between students and instructors and among students was 
supported via several technologies. Survey results indicated that 
off-campus students found delivery of lectures via video technology 
(two-way video, one-way video with two-way audio, broadcast video, 
and videotape) to be acceptable. Students felt that the quality of 
their mediated interaction with their instructor was as good as or 
better than experiences in face-to-face classes, and half the 
students felt that the frequency of their interactions were the same 
or more frequent. To promote interaction between faculty and students 
or among students, technologies such as audio-conferencing, 
electronic mail, or voice mail were more effective than video 
technologies. Faculty felt that the technologies that let them act 
the same way they do in face-to-face classes provided the best tools 
for interaction. The technologies are pushing changes in the 
relationship among course content, mentor, and student. (Contains 18 
references . ) (JDD) 
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New Pathways to a Degree: 
An Assessment of the Use of Instructional Technologies 
at Seven institutions 



Abstract: 



Most higher education institutions have invested, to at least some degree, in learning 
technologies that open the campus-from computers to satellites. However, we know 
very little about whether the benefits of these technologies justify their cost to 
institutions (e.g., How do they affect a student's ability to learn, to perform in the 
learning environment, or to attain educational goals?) This paper reviews the two-year 
evaluation of the seven Annenberg/CPB New Pathways to a Degree electronic learning 
projects. The evaluation's design and findings will be discussed; and the applicability of 
the evaluation design to other institutions and lessons learned about evaluating 
technology based learning programs will be examined. 
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American business and industry, in urban or rural settings, must find new ways to 
constantly increase and reinforce the skills of their employees. The enormous complexity of 
modern technology and the speed at which new innovations arise makes career-long training a 
necessity (Baldwin, 1991; see also Reich, 1991b; Commission on the Skills of the American 
Workforce, 1990). The inflexibility of the traditional university schedule, however, often presents 
a hardship for this clientele (Hall, 1991; Carey, 1991). 

The use of new technologies plays an important role in higher education's overall strategy 
for meeting the needs of rural communities, and the life-long learning needs of the workforce. 
Through the use of telecommunications and computers institutions of postsecondary education 
have found new ways to deliver courses to the "new majority learner". Moreover, studies 
conducted on the instructional benefits of electronic learning technologies suggest that they have 
the potential for changing the relationship between teacher and student. Telecommunicated 
discussions and computer simulations put everyone on a more equal footing regardless of 
assertiveness, or physical, regional, and cultural differences. Moreover, learning becomes a 
process of understanding and application rather than memorization, in which students are 
transformed from passive receptors of knowledge to active participants in the learning process, 
and the instructor becomes a mentor or facilitator rather than a purveyor of knowledge (Kozma 
and Johnston, 1991; Johnstone, 1991). 

If the full potential of these technologies is to be realized, however, many issues remain to 
be resolved. First, while there are many excellent examples of innovative instructional 
programming, computers and telecommunications are not being used to their full potential in the 
classroom. While higher education institutions have invested heavily in telecommunications and 
computing equipment, the teaching process has not changed to take advantage of these 
technologies, and only a small proportion of the faculty use technology in instruction (Kozma and 
Johnston, 1991; Carey, 1991). Second, questions remain to be answered regarding the 
effectiveness of computer-aided learning modules and telecommunications in instructional 
programs. It is not clear from the available research, which focuses on academically advanced 
and highly motivated students, whether these technologies succeed as well with "at-risk" and less 
motivated students. Moreover, while students give telecommunicated courses high marks, the 
majority say they prefer to take classes where they meet face-to-face with the instructor and other 
students (Johnstone, 1991). 

The growing interactivity of the medium may shift these preferences somewhat, but more 
research is needed on the best instructional uses of face-to-face and telecommunicated 
interaction. Unfortunately, an adequate body of research does not exist on the effectiveness of 
technology based learning programs. According to Kozma and Johnston (1991), very few of the 
EDUCOM/NCRIPTAL innovative computer-based learning programs include an assessment plan. 
There are barely enough resources to support program development, and most faculty involved in 
these programs are not trained in assessment methodologies. Institutional researchers, through 
their knowledge of assessment methodologies, can contribute a great deal to our understanding of 
this area. 

This paper reviews the two-year evaluation of the seven Annenberg/CPB New Pathways 
to a degree electronic learning projects. The evaluation's findings will be discussed, and the 
applicability of the evaluation design to other institutions and lessons learned about evaluating 
technology based learning programs will be examined. 
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New Pathways To A Degree: Overview 

The New Pathways to a Degree initiative of the Annenberg/CPB Project provided $1.5 
million to seven projects. 1 Each of the funded projects was selected because they proposed 
innovative ways to use technology to create new educational opportunities for "new majority 
learners." New majority learners are defined as students with jobs, home responsibilities, busy 
schedules, and other life circumstances which make it extremely difficult for them to attend 
college in conventional ways. 

The seven projects grouped by major characteristic are: 

Multi-institutional projects: 

BACCALAUREATE DEGREES AND STUDENT SERVICES: USING THE NEW 
TECHNOLOGIES TO EXPAND ACCESS, The Oregon State System of Higher 
Education and Oregon Ed-Net. Its self-stated goal was to develop a model for how 
schools can offer complete baccalaureate degree programs through a statewide 
educational network. 

PROJECT BREAKTHROUGH, West Virginia Higher Education System, 
Morgantown. Its self-stated goal was to develop a model for a collaborative statewide 
coalition that will make undergraduate degree opportunities available to rural adults. 

COMMUNITY COLLEGE OF MAINE, University of Maine at Augusta, and the 
Maine Community College System. Its self-stated goal was to build on its current 
system of linking all public higher education institutions with off-campus sites such as 
public schools and off-campus centers to offer a coherent sequence of courses for an 
associate of arts degree to rural learners in the state. 

New Programs, New Populations: 

ELECTRONIC ACCESS TO WEEKEND COLLEGE, The College of St. Catherine, 
St. Paul, Minnesota. Its self-stated goal was to serve as a model for how 
colleges-particularly those with weekend programs-can use technology to remove 
educational barriers and meet the needs of the adult student. 

COMMUNITY LEARNING NETWORK, Indiana University-Purdue University at 
Indianapolis. Its self-stated goal was to develop a model for how urban universities can 
provide access and community support for minority students. 



The Annenberg/CPB Project received 243 proposals of which they were able to fund seven. Because of 
the broad interest in the project and the high quality of the proposals, the Annenberg/CPB Project decided to 
support additional related activities and invited 32 of the best applicants to become "New Pathways Associates." 
The Annenberg/CPB Project conducted on-campus workshops, supported three annual meetings and several 
electronic discussion groups, which enabled the Associates to join with the funded institutions in the ongoing 
discussion which surrounded New Pathways to a Degree. 
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Technological Pragmatists: 



ENHANCED ACCESS TO LEARNING THROUGH TECHNOLOGY, Rochester 
Institute of Technology, Rochester, New York. Its self-stated goal was to develop an 
affordable model for using technology to offer highly accessible upper division courses 
to give students even more freedom to choose where, when and at what pace they 
study. 

ACCESS, INVOLVEMENT AND SUCCESS IN DISTANCE LEARNING, The 
Extended Leaning Institute, Northern Virginia Community College. Its self-stated 
goal was to use public and cable television, computer conferencing, two-way 
compressed video, audio conferencing, voice mail and videocassettes to make it 
possible for students to earn an entire associate degree in general studies or business 
administration. 

The seven funded projects are remarkable in their diversity-both in their organization and 
in the activities in which they are engaged. Students in one or another of the New Pathways 
projects used just about every learning technology available: 

♦ Some students participate at home in a course broadcast live from an on-campus 
classroom. These students can "raise their hands" and participate in the 
classroom discussion by means of a return video, a return audio signal, or a 
dial-up telephone. 

♦ Some students learn by video tapes produced on campus or by commercial 
providers. These video tapes are distributed to students by mail or courier. 
Students can watch these videos at their own convenience rather than being 
bound by a specific broadcast time. The students speak to their instructor by 
toll-free telephone or other means. 

♦ Some students receive courses on a computer at home or at work-the course 
content is delivered to students as a series of computer files via a computer 
network. Students interact with the instructor and institution primarily by 
computer network. E-mail is used to support communication between teacher 
and students and between individual students, and electronic bulletin boards are 
used to support one-to-many communications — teacher to students, and student 
to students. 

♦ Some students participate in live classes via telephone-several speakerphones are 
connected to each other by an audio bridge: every participant can hear and be 
heard by the instructor and by other students connected to the bridge. 

♦ For any of the above models, students may use a remote learning center. The 
center, usually a dedicated room in a school, library, or community center, is 
equipped with a large array of communication and learning tools-video 
monitors, VCRs, fax machines, telephones, computers with modems for 
connecting a campus network, and computer peripherals like CD-ROM readers, 



printers, and so forth. The course may require students to use any or all of these 
communication devices. 

In most of these situations, students meet other students, face to face or electronically, and 
form learning communities. In all of these modes, students have alternative means of 
communicating with instructors outside of class hours--by toll-free telephone circuits, voice mail, 
computer network, fax, mail or combinations of these communication systems. 

Each of the New Pathways' projects used a different version of these models. Table 1 
offers a glimpse into the range of technologies used at each of the seven projects. It shows the 
variability between the projects with regard to the varieties of technologies that were available to 
the students and faculty. 

All classes at all of the projects used some form of print media, and most had access to 
some form of video technology. Electronic mail was also widely available for use by the students 
and faculty involved in these New Pathways classes. Moreover, all New Pathways courses had 
two things in common: (1) technologies are used in combinations, and (2) that communication 
between students and instructors, and among students are supported via several technologies. 
Assessment of the effectiveness of these projects focused on the effectiveness of course delivery, 
communication between instructor and students, the degree to which these technologies enabled 
successful teaching and learning, and how institutions adapted to the service needs of the faculty 
and students. 



TABLE 1 

Types of Technology used in New Pathways 
by Project 


Project 


Technologies 


All NP Classes 


Some NP Classes 


OJPUI 


Print, One-way video, voice mail, videotapes 
facsimile, e-mail 


Videodisc, application software 


Maine 


Print, one-way video/two-way audio 


Voice mail, audioconferencing 


Northern Virginia 


Print 


One-way video, videotape, voice mail, e-mail, 
two-way video, application software, 
audioconferencing 


Oregon 


Print, two-way vide > or one-way video/two-way 
audio 


E-mail 


RJT 


Print, videotapes, one-way video, e-mail, 
audioconferencing 


Application software, computer testing, picture phone 


St. Catherine 


Print, application software, e-mail, facsimile, 
audiotapes, videotapes 




West Virginia 


Print, one-way video/two-way audio, voice-mail 


Videotape, e-mail* computer networks 
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The Evaluation Process 



The Western Cooperative for Educational Telecommunications was selected by the 
Annenberg/CPB Project to conduct the evaluation of the New Pathways funded projects. The 
Western Cooperative staff acted as coordinators for individual evaluators who closely followed 
the seven projects. The evaluation was designed to: 

♦ assess the impact of telecommunicated instruction upon students; 

♦ determine the effect of technology and mixes of technology on student learning, 
instructional processes, organizational policies, and support structures; and 

♦ determine the overall impact of the program. 

In year one of the evaluation, each evaluator made at least one two-day site visit and 
maintained regular face-to-face and telephone contact with the on-site project teams, other 
evaluators, and the contractor's staff. They interviewed project staff, faculty, and students and 
collected baseline information on the project. In addition, each project was asked to complete an 
environmental scan to help the evaluation team identify the types of data available from individual 
project sites. In year two, each evaluator made a second site visit. During the site visit they 
interviewed students, faculty, and project staff about their experiences with and perceptions of the 
project. 

The evaluation process included a survey of a sample of faculty and students from each 
project. The survey provided additional insights into the anecdotal information collected and 
supported the generalizations across projects observed during the site visits. 

This paper focuses exclusively on the methodology and results of the formal survey 
evaluation. More information on the conclusions of the full evaluation process can be found in 
the final project reports: New Pathways to a Degree: Technology Qpens the College (Markwood 
and Johnstone, 1994a) and Seven Technology Stories: New Pathways to a Degree (Markwood, 
and Johnstone, 1994b). 



New Pathways Faculty and Student Surveys 

Evaluators visiting each project heard many comments about how students and faculty 
worked with these various technologies. To verify anecdotal information about student and 
faculty experiences gathered from these site visits, surveys of students and faculty were 
conducted. The following section discusses the survey design and methodology, and reviews the 
results of the student and faculty surveys. 

The student and faculty in the seven New Pathways projects were administered during the 
Spring 1993 academic term. The student and faculty surveys were designed so thai comparisons 
could be made between faculty and students and across programs and, where applicable, between 
on-campus and off-campus students. 



Sampling. 



New Pathways courses at each program were the sampled population. Each of the seven 
New Pathways projects was asked for a list of the New Pathways courses taught during the 
Spring 1993 term. These lists included course name and number, total enrollment attending at on- 
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and off-campus locations (where relevant), and the technologies available for use in the course. 
Table 1 shows the range of technologies used in the New Pathways courses and the percentage of 
students with access to each. 



TABLE 1 

Percent of New Pathways Students with Access to 
Specific Technologies 


Technology 


Off-campus students 


On-campus students 


Videotapes 


70% 


67% 


Audio conferencing 


42% 


46% 


Televised lectures 


41% 


51% 


Electronic mail 


38% 


20% 


Audio talk-back 


36% 


40% 


Voice-mail 


29% 


19% 


Audiotapes 


12% 


20% 


Two-way video 


11% 


15% 


Computer programs 


10% 


15% 



Courses were selected based on their field and discipline, enrollment, and mix of 
technologies. The goal was to include a representative mix of courses from each program. In 
programs offering only a few courses, all courses for the Spring 1993 term were included in the 
sample. Overall, 35 New Pathways courses were sampled (See Table 2). Of these 13 had 
students simultaneously enrolled on-campus and at off-campus sites. 



TABLE 2 

Survey of Students and Faculty 
Courses Sampled in Spring 1993 


PROGRAM 


COURSES SAMPLED 


TOTAL No. 
of COURSES 


IUPUI 


2 


2 


Maine 


6 


8 


Northern Virginia 


11 


16 


Oregon ' 


3 


4 


RTF 


8 


8 


St. Catherine 


2 


2 


West Virginia 


3 


3 


TOTAL 


35 


43 



As Table 3 shows, the courses surveyed vary in size from 4 to 304 students. The larger 
courses tended to be one-way video, two-way audio courses that we refer to as "video with audio 
talk-back." The smallest courses were computer-based classes. 



TABLE 3 
Number of Students Enrolled 
in Sampled Courses 


MEAN 


51 


MEDIAN 


35 


RANGE 


4 to 304 



The technologies used also ranged widely. While all the technologies listed in Table 4 
were used in the classes represented in our survey, some classes used more than one technology. 



TABLE 4 
Technologies used in Sampled Courses 


Specific 
Technology 


Number of 
Courses 


Percent of 
Students 


Videotapes 


18 


- 56% 


Computer Conferencing 


19 


48% 


Computerized Instruction 


13 


45% 


Televised Lectures 


11 


35% 


Video with Audio Talk -back 


9 


29% 


Voice Mail 


8 


20% 


Two-way Video * 


2 


11% 


Audio Conferencing 


9 


18% 


Audiotapes 


2 


7% 


* one course dropped from analysis due to non-response 



As Table 5 shows there is a wide range of academic disciplines represented by the courses 
surveyed. Courses were selected across a wide range of disciplines to smooth out the effects 
specific to particular teaching practices (e.g., mathematics versus literature). 



TABLE 5 

Percent of Sampled Courses by Discipline 


Humanities 


17% 


Art/Drama 


9% 


Science 


17% 


Health 


6% 


Social Science 


11% 


Language 


3% 


English 


11% 


Math 


3% 


Computer 


11% 


Vocational Education 


3% 


Business 


9% 


TOTAL 


100% 



d 7 in 

ERIC 10 



Survey Design And Dissemination. 



Student Survey 

The student surveys were designed to measure student satisfaction and proficiency levels 
with, and use of individual technologies; satisfaction with the course, instructor, and support 
services; and students' academic goals and achievement levels. The full student questionnaire 
contained 142 questions in 12 individual sections. It was designer to be easily tailored to the types 
of instructional technologies 2 available for each course. Sections C through K of the survey were 
included only if the technology asked about was available for use in that course. Individual course 
questionnaires varied in length from 39 questions, to a maximum of 86 questions. 

The survey contained a mix of likert-scale questions and open-ended questions. All 
students were asked for information about their academic background, general course information 
and, with the exception of the University of Maine 3 , were asked to evaluate their faculty. Students 
also were asked about their experiences and satisfaction with the specific technologies used in 
their course and were given the opportunity to comment on the advantages and disadvantages of 
using each technology. The name of the sampled course appeared at the top of page one or the 
questionnaire. At several points in the survey, students were directed to answer the questions 
with this specific course in mind. 

The distribution of the student questionnaires was timed so that students would receive 
them shortly after mid-term. This gave students enough time to be familiar with the course and 
the technologies used, and to have earned enough grades on assignments and exams to give us 
informed responses. Since beginning and ending dates, and term length vary across institutions, 
the timing of the questionnaire distribution was staggered. 

Each program took a role in distributing the questionnaires and conducting follow-up. 
Each program was sent the appropriate number of questionnaires and cover letters for each 
course sampled. The program office than took responsibility for distributing the questionnaires, 
conducting follow-up activities, and collecting the responses 4 Some programs mailed the 
questionnaires directly to the students, while others sent them to the faculty who distributed them 
in class or at distance-learning sites. The choice of distribution was up to the program and was 
based on their perception of the best way to reach their students. 

Each of the programs were asked to conduct follow-up activities in order to ensure a high 
response rate. The particular type of follow-up activity varied across programs depending on the 
resources they had available. At least two programs sent out reminder postcards or letters to 
students (Northern Virginia, RIT). Indiana University-Purdue University at Indianapolis (RJPUI) 
also sent out reminders by mail to one site-based course. In the majority of cases, students were 
reminded by faculty/proctors at the remote site to complete and return the questionnaire. 

The overall project response rate was very acceptable. A total of 1,890 questionnaires 
were sent to the 7 programs; 552 questionnaires were returned as extra or undeliverable, leaving a 
total population of 1,338 students. Five hundred-fifty completed qu estionnaires were returned for 

The types of instructional technologies covered in the survey were videotapes, audio conferencing, 
televised lectures/one-way video, electronic-mail/computcr conferencing, video with audio talk-back, voctc-mail, 
audiotapeds, two-way video, and stand alone computer programs/courscswarc). 

The faculty union contract at the University of Maine prohibits conducting faculty evaluations without the 
direct involvement of the faculty in the design of the questionnaire. 

Students in the Oregon program were asked to mail their completed surveys directly to the Western 
Cooperative for Educational Telecommunications. 

- / 

8 

11 




a response rate of 41.1 percent. Response rates varied across projects, ranging from a 13 percent 
overall response rate at Indiana University-Purdue University Indianapolis (IUPUI) to a 100 
percent response at the College of St. Catherine (see Table 6). Analyses of the total student 
population and the total distance learning population were weighted to account for these 
differential response rates. 



TABLE 6 

Response Rates by Project and Type of Student 







ALL STUDENTS 


OFF-CAMPUS STUDENTS 




Distributed* 


* Returned 


Response Rate 


Distributed* 


Returned 


Response 
Rate 


No. VA 


177 


105 


59.3% 


177 


105 


59.3% 


Maine 


366 


207 


56.6% 


282 


145 


51.4% 


IUPUI 


324 


43 


13.3%+ 


31 


7 


22.6% 


West VA 


164 


86 


52.4% 


103 


50 


48.5% 


Oregon 


107++ 


51 


46.7% 


67 


27 


40.3% 


RIT 


192 


50 


26% 


192 


50 


26% 


St. Kate's 


8 


8 


100% 


8 


8 


100% 


TOTAL 


1,338 


550 


41.1% 


860 


392 


45.6% 


* The number distributed is equal to the number of questionnaires sent to the program less those that were extra 
or undeliverable 



* The relatively lower response rate for IUPUI is probably due to several factors: (1) students at IUPUI completed 3 
questionnaires and a 30 minute interview earlier in the semester and were reluctant to take time to complete another 
questionnaire; (2) a large on-campus section of one New Pathways course in which the students had little stake in 
the outcome of the evaluation was included in the IUPUI sample (if this course is excluded the response rate 
increases to 22.5 percent). 

" One course in the Oregon program was dropped (Anthro 31 9) from the analysis because questionnaires were 
no; distributed to all students in the course. 



Faculty Survey 

The faculty survey was designed with crosswalks to the student survey. Faculty were 
asked comparable questions about their satisfaction and perceived proficiency levels with 
individual technologies; satisfaction with available support services; perceived differences between 
dynamics in distance learning and regular classroom settings; and the effect of involvement in 
distance learning curriculum on their career development. They also were asked to answer select 
questions about their academic and professional background taken from the National Center for 
Education Statistics' 1992 National Survey of Posisecondary Faculty (NCES, 
forthcoming) --zntioXmg a comparison with the professoriate as a whole. 5 The final survey 
included 79 questions in 4 sections. Like the student survey it included a mix of likert-scale 
questions and open-ended questions. 

All faculty teaching courses in the seven projects during the Spring 1993 term were sent 
questionnaires. Distribution was timed so that all faculty would receive the questionnaire prior to 
the end of the term. A total of 40 questionnaires were sent to faculty, 3 1 were returned for a total 
response rate of 78 percent. Comparisons of faculty responses across projects were not possible 



Results from the 1992 National Survey o/Postsecondary Faculty arc not yet available. 
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for two reasons: (1) to protect the confidentiality of participants, and (2) because only a small 
number of faculty from each project were included in the sample. 



Survey Results 



The following discussion of the survey results focuses on two areas that are of particular 
concern to college and universities that serve "new majority students" and/or that operate 
technology delivered learning programs. First, we discovered several differences between the 
responses of off- and on-campus students that may assist colleges and universities design more 
effective means of delivering academic support and student services to "new majority" students. 
Second, faculty and students responded to several sets of questions designed to measure how 
these technologies effect class dynamics. It is frequently assumed that technology delivered 
courses lack meaningful interaction between students and faculty, and among students. 
Responses by faculty and students in the New Pathways program may help to dispel that myth. ' 

The "New Majority" Student 

Providing academic and student services to students studying at off-campus presents 
admissions officers, academic advisers, and student service administrators with a unique 
challenge. On-campus facilities designed to meet the needs of the on-campus student-the 
counseling center, registrar, student union, library, and other facilities-are not accessible and may 
not be as useful to the "new majority student" studying at a distance. One of the first issues 
institutions must deal with when serving these students is what information and services they 
need, and how to make these services available to them. 

Distance from campus is not the only reason students have for choosing to study 
off-campus. In general, off-campus students participating in the New Pathways program tend to 
be involved in several things at once; including working a full-time job and raising a family. In 
the programs which serve rural populations, off-campus students (68 percent) tend to live 
further--20 miles or more-from campus than on-campus students (26 percent) However, in the 
programs which serve urban populations, off-campus students (83 percent) are more likely to live 
closer to campus-within 20 miles-than on-campus students (26 percent). Convenience, not 
distance, seems to be the single greatest reason these students choose to take a course via 
technology. Indeed, the most common response given by all of the off-campus students when 
asked about the advantages of taking a technology delivered course was that it was offered at a 
convenient time or location, or allowed them to complete course work at. a pace and at times they 
control 1 d. 

In many ways the off-campus students are more motivated than the on-campus students. 
The off-campus New Pathways' students are ambitious and expect to succeed. Off-campus 
students anticipate receiving higher grades than do the on-campus students. Ninety-three percent 
of off-campus students say they expect to receive a B or better in "this course", compared to 84 
percent of on-campus students. Further, most of these off-campus students see a two-year degree 
or certificate as an immediate goal, not necessarily a final stopping place. While off-campus 
students are less likely to say they expect a degree from "this institution", almost equal 
percentages of off- (94 percent) and on-campus (98 percent) students eventually expect to 
complete at least a bachelor's degree (see Figure 1). 
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Figure 1 
New Pathway's Student Survey 
Highest Degree Expected from Any institution 
On-campus and Off-campus Students 
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Technology is viewed by these students as a tool which enables them to achieve their 
educational objectives. In general, they were always more positive about technology than were 
on-campus student. Less than 5 per cent of the off-campus students "disliked" any of the specific 
technologies. And, with the exception of voice-mail, a majority of off-campus students said they 
enjoyed using each of the technologies and would take another course that used them. 
Off-campus students (19 percent) also are more likely than on-campus students (1 percent) to say 
they chose this course because it was "offered via technology" (see Figure 2). 



Figure 2 
New Pathways' Student Survey 
On-campus and Off-campus Students 
Decided to Take This Course Because: 
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In some respects off-campus students did not vary much from on-campus students. Only 
the off-campus students learned about courses from the program pamphlet and the newspaper. 
However, the majority of off- (63 percent) and on-campus (81 percent) students say they learned 
about the availability of this course through the college catalog (see Figure 3). Further, while 
more off- (27 percent) than on-campus (15 percent) students reported that the they decided to 
take courses at this time "to advance in their current position", majorities of both groups (58 
percent off-campus; 74 percent on-campus) say they are taking courses because they want to 
"earn a college degree" (see Figure 4). 



Figure 3 
New Pathway*' student Survey 
On-campus and Off-Campus Students 
Learned thk Course wae Available From: 
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Figure 4 
New Pathways* Student Survey 
Reasons for Taking College Courses for 
On-campus and Off-campus Students 
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Class Dynamics 

These projects presented us with the unique opportunity to evaluate the perceived effects 
of various instructional technologies on the "classroom" experience. There is more to a college 
course than absorbing the words of a faculty member. A critical part of any effective college 
course is the opportunity for students to discuss ideas among themselves and with the faculty. 
Indeed, much of the literature on learning emphasizes the importance of actively engaging 
students in the learning process (Cheney, 1990; Task Group on General Education, 1988; Faculty 
Teaching Excellence Program, 1988; Boyer, 1987; Newman, 1985; Study Group on the 
Conditions of Excellence in American Higher Education, 1984). 

To measure the perceived effects of each of these technologies on class dynamics students 
and faculty were asked about the quality and quantity of interaction and the usefulness of each 
technology to learning the subject matter in "this course." Students and faculty were asked to 
compare their experiences with faculty/student and student/student interaction in "this course" 
with , those they have had in previous face-to-face classes. It should be remembered that this is a 
measure of perception only, and that there is no control for the diversity of prior experiences 
students and faculty have had in their face-to-face classes. Nevertheless, it does give us some 
insight into the way students and faculty perceive their interactions in technology medicated 
courses. 
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Student/Instr u ctor Interactions . The anecdotal evidence gathered by the site evaluation 
team suggested there may be both quantitative and qualitative differences between mediated and 
face-to-face interaction in these courses. Therefore, we asked the students about both the quality 
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and frequency of their interactions with their instructor. Overall the students thought that the 
quality of their mediated interaction was at least as good as, or even better than, the experiences 
they had in face-to-face classes. It also is interesting to note that approximately one-half thought 
the frequency of their interactions were the same as, or more frequent than those in a face-to-face 
situation. 

Of course, students reacted to these questions differently depending on the type of 
technology in question. 6 More than three-fourths of the students rated the quality of interaction 
with their instructor using audio conferencing, electronic mail, and voice-mail as equal or superior 
to face-to-face classes, compared with only 62 percent of those using audio talk-back (see Figure 
5). Moreover, more than one-half (57 percent) using electronic-mail, nearly one-half (48 percent) 
using audio-conferencing and voice-mail, and approximately two-fifths (44 percent) of those using 
audio talk-back rated the frequency of their interaction with the instructor as the same or more 
frequent than in previous face-to-face courses. 




Figure 5 
New Pathways' Student Survey 
Compared to Face-to Face Classes, 
Quality of Interaction with Instructor 
is Unchanged or Improved by use of: 




Since the respondents to each technology section are for all practical purposes separate 
samples, the only way to evaluate the significance of the difference between reactions to audio 
talk-back and the other technologies was with a nonparametric chi-square. We tested the 
assumption that if the type of technology has no effect on student/faculty interaction the 
distribution of responses to frequency and quality of interaction for electronic-mail, voice-mail, 
and audio-conferencing would be equal to the distribution of responses for audio talk-back. The 
sample size for voice-mail was too small to yield statistically meaningful results, and the difference 
between responses regarding the frequency of interaction with audio-conferencing and audio 
talk-back was not statistically significant. However, the perceived quality and frequency of 



This analysis includes comparisons between four of the five interactive medium included in the 
evaluation: Audio talk-back, audio conferencing, electronic-mail, and voice-mail. Due to a small sample size and 
poor response rate an analysis of responses regarding the use of two-way video arc not included. Only two 
two-way video courses were being offered t the seven projects, and one course was dropped from the analyses due 
to a poor response rate. 
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interaction with faculty was statistically significant at the .001 level with chi-squares of 56.9 and 
1 1.8, respectively. 

If one considers the behaviors involved in using these different technologies, these 
differences in the student's reactions are not surprising. Audio talk-back is the aspect of a 
one-way video, two-way audio set-up designed to allow students to communicate with an 
instructor they can see and hear but who cannot see them. The first group of technologies 
(electronic-mail, voice-mail, and audio-conferencing) each requires some planning on the part of 
the instructor that is aimed specifically at facilitating interaction with his/her students. Audio 
talk-back is typically used in conjunction with a televised lecture. This means there may, or may 
not, be any rethinking of the way in which interaction is accomplished between the instructor and 
the students. This same argument holds true for two-way video classes. While the responses we 
have from students using two-way video is insufficient to be included here, they do seem to be 
closer to the audio talk-back responses than those relating to the other technologies. 

Electronic mail is the only technology for which more than one-half of students rated the 
frequency of interaction equal or superior to face-to-face classes. This again may be a function 
of the behaviors associated with the use of these technologies. In a face-to-face class, each 
student competes for the chance to interact with the instructor during a limited amount of time. 
The same is usually true in an audio-conferencing environment. Even though voice-mail and 
electronic-mail are free from time constraints, only electronic-mail gives instructors and students 
the flexibility to think about and compose a detailed message. For users of electronic-mail, this 
may encourage more frequent, high quality exchanges. 

To determine whether the instructors and the students were equally satisfied with the 
capabilities and practices of interacting with one another via technology, we also asked the faculty 
to compare their mediated experiences with those in the same classes they had taught in a 
face-to-face environment. They were asked to rate both the frequency and the quality of 
interaction. We also asked them to write their answers to the question: "What differences have 
you noticed between the frequency and quality of interaction in courses using instructional 
technology versus more traditionally taught courses." 

Interestingly, the faculty's perceptions and the students' don't agree. While the students 
thought the quality of interaction with their instructors was unchanged or improved least often 
with one-way video/audio talk-back technology, the instructors rated the quality of their 
interactions with students highest with one-way video/audio talk-back technology. One hundred 
percent of those using one-way video with the audio talk-back component thought their 
interactions with students was at least the same as that they experienced in face-to-face classes. 
This tends to support an earlier argument that faculty need not radically rethink how they teach 
their class when they use this technology and consequently perceive it as about the same. 
Whereas the other three technologies require that faculty consciously change their teaching 
strategies. These other technologies still rated very high with regard to the quality of interaction 
being considered as good as, or better than the face-to-face class experiences 7 (see Figure 6). 
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The number of faculty responding to questions about each of these technologies is about the same (8 to 15 
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Figure 6 
New Pathways' Faculty Survey 
Quality of Interaction with Students Compared 
to Face-to-Face Classes is 
Unchanged or Improved for: 



ERLC 



In response to the open-ended question several faculty mentioned changes in the mode of 
interaction without really commenting on how it was different. For example, according to one 
instructor, M [s/he] spend[s] more time interacting on the phone and in writing than in person." 
Others reflect the qualitative effect of the media. The following examples are characteristic of the 
responses from faculty using real-time video as the primary means of communicating with 
students: 

♦ "The discussion stays on task on the air. No digressions. The listening skills 
improve since students focus more on the call-in comments." 

♦ "Students need greater encouragement to offer feedback to instructor." 

♦ "Interaction is less spontaneous." 

♦ "There is much less interaction. Students call voice mail with questions." 

♦ "It was much easier to initiate the interaction in a classroom course than a 
televised lecture course." 

These are the same types of comments that instructors using this technology have been 
making for the past ten years. This is probably why many institutions are going beyond the "on 
the air" interaction as the only means for communication. Faculty using other technology (such 
as, electronic-mail, voice-mail and audio-conferencing) to interact with their students say things 
like: 

♦ "Role of instructor changes from being all info/concept provider to greater 
emphasis on working with students to use the info/tech." 

♦ "I have greater awareness of the individual students and his/her individual learning 
styles." 

♦ "I never felt 'connected 1 with the students, mostly because I spent so much time 
energy and 'stress 1 attempting to understand the computer." 
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♦ "Quality - more in-depth, one-on-o;\e... This [e-mail] does force an instructor to 
address the individual differences. It is more obvious that learners are at all 
different levels of achievement... Electronic group interaction can never replace 
the face-to-face situation." 

♦ "Phone lines are not great for assessing pronunciation. On the other hand, the 
students receives feedback on every word they write or speak - highly 
personalized feedback." 

♦ "Voice mail does give students unlimited access to instructor." 

These comments are characteristic of the faculty responses. They highlight the differences 
between what students and faculty do when they communicate using voice-mail, electronic-mail, 
or audio conferencing. These technologies offer more intimate channels of communication, but' 
also require that faculty learn new skills, and invest more time in course preparation. As a result, 
they tend to change the teaching/learning environment more radically than do video-based 
technologies. 

Student/Student Interaction. As previously mentioned, a good quality learning 
environment should include more than just information exchanged between the student and the 
faculty member. The college student on a campus also learns from his/her peers and from the 
opportunity to discuss new ideas. We wanted to know if any of these technologies seemed to be 
a superior tool for this function. It is interesting that even though more than one-half of the New 
Pathway students thought the quality of electronic-mail (82 percent), audio-conferencing (74 
percent) mediated interactions were at least as good as face-to-face classes for student to student 
interactions, while a somewhat smaller percent (67 percent) thought audio talk-back was. The 
reason this is interesting, is that audio talk-back technology is usually used in a peer group setting. 
That is, groups of students gather at remote sites to participate in the televised class session. 
Under most circumstances, voice-mail, audio conferencing, and electronic mail are used by 
isolated students, yet they still appear to provide adequate opportunities for peer interaction (see 
Figure 7). 
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Figure 7 
New Pathways' Student Survey 
Compared to Face-to Face Classes, 
Quality of Interaction with Other Students 
is Unchanged or Improved by use of: 
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Other Cha nges in Class Dynamics . Faculty were asked two questions specifically about 
differences in class dynamics between New Pathways courses and traditionally taught courses: (1) 
Were there differences in the classroom dynamic?, and (2) To what do you attribute those 
differences? These responses are among the most difficult to generalize. There seem to be 
differences attributable to institutions, to technologies used, to individual teachers, and to 
students. One response may appropriately set the tone for this discussion: 

"I don't believe this is a good or necessary comparison. This is a different means of 
learning; it works for some students and not for others. Making this comparison for 
my course doesn't, accomplish anything." 

Faculty had very different responses to the issue of changes in class dynamics. Those who 
were positive about the changes in classroom dynamics mention being better prepared, having 
better graphics, the greater variety of supporting materials, the superiority of graphics presented 
by the new technologies as compared to the overhead projector, that students are less inhibited, 
that there is more interaction, and that they get more information on how students are performing. 
Those who were negative about the changes in classroom dynamics mention that students are 
generally more hesitant to interact and tend not to attend class sessions (they may be available on 
videotape for later viewing), some students become more invisible (others are drawn in), there are 
fewer questions, more lecture, more structure, less free inquiry, less fun, there is no comparison 
between face-to-face and television. 

In general the improvements are caused by the demands of the technologies-to be better 
prepared, to have thought through the instructional strategy from beginning to end, and to have 
found and presented the appropriate materials needed to support the course. The negative 
impacts on the course dynamics on the other hand are caused by the distance between student and 
teacher and by personalities of both teachers and students. 



Limitations Of The Evaluation Process 



One of the most obvious limitations of this evaluation is that the students taking these 
New Pathways courses are self-selecting. These were students who had reasons for not being on 
a campus, yet were motivated to seek out educational opportunities. They had high expectations 
of themselves, were serious about their studies, agenda oriented, and grateful for the opportunity 
to have access to a college education. Consequently, they were willing to try new experiences and 
successfully cope with the difficult time-management commitments required. Although 
information on course retention rates is available from only a few of the projects, their data 
suggest that course completion rates were very high (90 percent or higher). 

These students also may be naturally high achieving individuals, with previous experiences 
that shaped their interest in technology-based instruction. Since we had no way of controlling 
from previous educational or technological experiences, we don't know how the skills and 
experiences they brought with them influenced their perception of this experience. 

Although the faculty also seem to be self-selecting, at least some instructors indicated they 
were teaching the course because it was required of them. Nevertheless, the sample size is too 
small to make generalizations from the faculty responses. 

Finally, time constraints made it impossible to conduct follow-ups to see how this 
experience affected students and faculty in subsequent courses, at work, or in other activities. 
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However, to varying degrees, on-going evaluation processes were initiated at each of the seven 
projects. While they do not provide consistent and comparable cross-project information, they X 
will provide some longitudinal information on the experiences of students involved in these 
distance learning projects. 

Summary And Conclusion 

This discussion has covered only a small part of the two-year evaluation of the seven 
Annenberg/CPB's New Pathways to a Degree projects. Nevertheless, the material presented here 
suggests that New Pathways students and faculty, in general, responded very well to all the 
technologies examined here. Even though we may not be able to generalize the responses of the 
New Pathways students to all potential college students, their responses have helped us learn a 
little about what types of technologies seem to work best for different types of tasks that are 
typically part of the "student experience" on- or off-campus. If a faculty member wants to 
"deliver" a lecture to a group of students via any of the video technologies (two-way video, 
one-way video with two-way audio, broadcast video, videotape) that seems to be acceptable to 
students studying off-campus. If the faculty member wants to have interaction with students or 
between students, then the video technologies seem less effective than all the other technologies 
used in these projects. According to the students, audio-conferencing, electronic mail, or 
voice-mail work equally well for student/faculty discussions. For student to student discussions 
electronic-mail seems superior. 

Most surveyed Naw Pathways faculty thought the technologies that let them act the same 
way they do in face-to-face classes provided the best tools for interaction. It is interesting to note 
that several faculty members surveyed indicated that the experience of mediated teaching does 
not compare at all with a face-to-face environment. It also is critical to note that all these faculty 
represent the first generation usin ; these new tools and the tools themselves are pushing changes 
in the relationship between course content, mentor, and the student. One faculty member even 
mentioned that he felt more like a mentor than a teacher. The tools themselves and the pedagogy 
surrounding them are evolving. It is hard to tell right now how all this can work and will work in 
the near future, but we do know that the technologies used in these seven projects all worked for 
most of the students who used them. 

The lessons learned from the New Pathways evaluation may be of use to other institutions 
exploring the benefits and costs of instructional technology programs. Although the literature on 
educational assessment is voluminous, the assessment of technology-based instructional programs 
is still in its infancy. A variety of assessment inventories are available for assessing satisfaction, 
educational and social behaviors, and perceptions of outcome. 8 Unfortunately, all of these 

* The Clearinghouse of Higher Education Assessment Instruments (1993) lists hundreds of instruments 

designed to measure everything from cognitive abilities and skills to affective measures such as feelings about 
interpersonal relations and values. Among these are: the Learning and Study Strategies Inventory (LASSI) 
designed to assess strategies and attitudes successful students use in learning (NCHEMS, 1993); the Motivated 
Strategies for Learning Questionnaire (MSLQ) containing self-reported items addressing student beliefs about 
themselves and how they work (NCHEMS, 1993); The College Student Experiences Questionnaire (CSEQ) and 
Community College Student Experiences Questionnaire (CCSEQ) includes measures of involvement in college 
activities (social and educational) and self-reported learning practices (Pace, 1990); The National Center for 
Higher Education Management Systems' inventory includes questionnaires designed to measure staff and faculty 
values; the impact of the college experience on alumni; and the educational goals and satisfaction with services of 
entering, continuing and completing students, and alumni (NCHEMS, 1994); the American College Testing 
Service's inventory includes instruments measuring student satisfaction with services and the campus environment, 
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inventories start with the basic assumption that the respondents are participating in a traditional 
classroom experience. For example, the College Student Experiences Questionnaire (CSEQ) 
includes several sections on college activities that are relevant for both students in traditional 
classroom settings and those in technology-based or distance learning environments (including 
Library Experiences, Experiences with Faculty, Course Learning, Experience in Writing, and 
Science). However, because it is designed with the traditional classroom experience in mind, 
items are frequently included that are inappropriate or meaningless for technology-based courses. 
For example, students are asked how often they "took detailed notes in class" or "read something 
in the reserve book room or reference section" (Pace, 1990). Other questions relevant to the 
technology-based course are omitted, such as how often students "communicated with faculty 
and/or other students via E-mail", "requested that a book be mailed to you from the library via 
computer", or "participated in an audioconference with other students". Similar problems exist 
with each of the other assessment inventories. 

Since the student experience and the terminology used in technology-based courses are 
specific to that context, institutions that want to assess these programs must create their own 
assessment instruments. While these may be uniquely suited to the institutions needs, the 
institution has no way of comparing their programs or students with other programs and students 
at other institutions. Nor does it give us a means for generating normed responses that enable 
comparisons of different models of instructional technology use. Clearly more work needs to be 
done in the assessment of technology-based instruction. 



and self-reported measures of academic achievement (ACT, 1994); and the Educational Testing Service's Program 
Self-Assessment Surveys assess satisfaction with and evaluation of the college environment for students, faculty, 
and alumni (ETS, 1994). 
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